{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b21b469b",
   "metadata": {},
   "source": [
    "## 获取待提取特征的文件\n",
    "\n",
    "提供两种批量处理的模式：\n",
    "1. 目录模式，提取指定目录下的所有jpg文件的特征。\n",
    "2. 文件模式，待提取的数据存储在文件中，每行一个样本。\n",
    "\n",
    "### 参数说明\n",
    "\n",
    "1. sample_dir： 你自己的样本目录\n",
    "2. model_root：你自己的模型目录，注意这里不需要精确到viz目录\n",
    "3. feature_name：你自己喜欢的层的名称，。\n",
    "\n",
    "# 注意：目前支持可视化功能的只有Resnet系列，其他的模型仍在测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8dee942",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from glob import glob \n",
    "\n",
    "sample_dir = r'数据目录'\n",
    "model_root = r'你自己的模型目录，注意这里不需要精确到viz目录'\n",
    "feature_name = r'avgpool'\n",
    "samples = glob(os.path.join(sample_dir, '*'))\n",
    "samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26847144",
   "metadata": {},
   "source": [
    "## 确定提取特征\n",
    "\n",
    "通过关键词获取要提取那一层的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e8d607a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from onekey_algo import get_param_in_cwd\n",
    "from onekey_algo.custom.components.comp2 import extract, print_feature_hook, reg_hook_on_module, \\\n",
    "    init_from_model, init_from_onekey\n",
    "\n",
    "model, transformer, device = init_from_onekey(os.path.join(model_root, 'viz'))\n",
    "for n, m in model.named_modules():\n",
    "    print('Feature name:', n, \"|| Module:\", m)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87f29370",
   "metadata": {},
   "source": [
    "## 提取特征\n",
    "\n",
    "`Feature name:` 之后的名称为要提取的特征名，例如`layer3.0.conv2`, 一般深度学习特征提取最后一层，例如`avgpool`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "541bfc3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "import os\n",
    "\n",
    "os.makedirs('features', exist_ok=True)\n",
    "with open('features/feature.csv', 'w') as outfile:\n",
    "    hook = partial(print_feature_hook, fp=outfile)\n",
    "    find_num = reg_hook_on_module(feature_name, model, hook)\n",
    "    results = extract(samples, model, transformer, device, fp=outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d3cbd6f",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7ac1309",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "features = pd.read_csv('features/feature.csv', header=None)\n",
    "features.columns=['ID'] + list(f\"DL_{c}\" for c in features.columns[1:])\n",
    "features.to_csv('features/feature.csv', index=False)\n",
    "features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9634e5f2",
   "metadata": {},
   "source": [
    "### 深度特征压缩\n",
    "\n",
    "深度学习特征压缩，注意压缩到的维度需要小于样本数\n",
    "\n",
    "```python\n",
    "def compress_df_feature(features: pd.DataFrame, dim: int, not_compress: Union[str, List[str]] = None,\n",
    "                        prefix='') -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    压缩深度学习特征\n",
    "    Args:\n",
    "        features: 特征DataFrame\n",
    "        dim: 需要压缩到的维度，此值需要小于样本数\n",
    "        not_compress: 不进行压缩的列。\n",
    "        prefix: 所有特征的前缀。\n",
    "\n",
    "    Returns:\n",
    "\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c649a10",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import compress_df_feature\n",
    "\n",
    "cm_features = compress_df_feature(features=features, dim=64, prefix='DL_', not_compress='ID')\n",
    "cm_features.to_csv('features/compress_features.csv', header=True, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1285696",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
